perf(amd): switch DSV4 ATOM AgentX to native RCCL DEP / 将 DSV4 ATOM AgentX 切换到原生 RCCL DEP - #2912
perf(amd): switch DSV4 ATOM AgentX to native RCCL DEP / 将 DSV4 ATOM AgentX 切换到原生 RCCL DEP#2912yhl-amd wants to merge 9 commits into
Conversation
将 DeepSeek-V4-Pro MI355X ATOM AgentX 的 c48 及以上测试切换到本地验证过的原生 RCCL DEP 配置，并保持低并发 TP 测试不变。 同步固定的 post-merge ATOM 镜像、EP8 元数据、关闭 TBO/EPLB、真实 MTP 接受率以及本地验证过的路由和 AIPerf 参数。
将 DeepSeek-V4-Pro RCCL DEP 性能变更记录中的占位链接替换为实际的 InferenceX PR 链接。
精简 DeepSeek-V4-Pro AgentX RCCL DEP 配置，移除与 CLI 或公共默认值重复的环境变量，并补齐本地验证使用的 3600 秒 warmup grace。
Restore the request-equivalent weight, prefill delayer, and decode interval requested for the AgentX run. Remove the newly introduced terminal MTP overrides while keeping the rest of the cleanup unchanged.
Keep the 3600-second agentic warmup allowance, but rely on the server keep-alive setting and AIPerf default benchmark grace period.
Remove the DEP-only 32K override so both TP and DEP retain the original 8192-token state checkpoint interval.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5c91ae8. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked the dropped AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID export in the DEP branch — the surrounding comment now only describes the dynamo-specific session-id header for DPA routing, so removing the generic fallback appears to be an intentional simplification rather than a regression.
Extended reasoning...
One confirmed finding (perf-changelog.yaml claiming zero request equivalence and a 32768-token checkpoint interval that contradict the script's ATOM_DP_LB_REQ_EQUIV=512 and hardcoded STATE_CHECKPOINT_INTERVAL_TOKENS=8192) is already queued for posting as an inline comment, so it is not restated here. I independently reviewed the script diff, confirming this discrepancy is real (STATE_CHECKPOINT_INTERVAL_TOKENS=8192 is set unconditionally at line ~57 with no path to 32768, and ATOM_DP_LB_REQ_EQUIV=512 is set explicitly for the DP_ATTENTION branch). Given a verified finding exists, approval is not appropriate. I additionally examined the other ruled-out candidate — the removal of AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID alongside AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID — and concluded it looks intentional: the updated comment describes only the dynamo-specific session header path for the new DEP transport, with no remaining reference to a generic fallback, consistent with dropping the redundant export. This is worth surfacing briefly as it was a real candidate concern that a reviewer might otherwise re-raise, but does not need extensive discussion. No new top-level architectural or security concerns were found beyond what's already captured in the inline finding.
| - "Move the concurrency 48/64/96/128/256 ATOM AgentX band from DPA with TBO and the default MoE transport to native RCCL DEP (TP8/DPA8/EP8), while preserving the TP-only concurrency 1/2/4/8/16 band." | ||
| - "Match the validated DEP run: disable EPLB and TBO, use least-tokens routing with session affinity and zero request equivalence, set the state checkpoint interval to 32768 tokens, and use real MTP acceptance on DEP." | ||
| - "Carry over the validated AgentX transport, terminal-MTP, and AIPerf timeout/failure settings for stable multi-turn replay." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2886 |
There was a problem hiding this comment.
🟡 (optional) The changelog entry's third bullet claims "zero request equivalence" and a "32768" token checkpoint interval, but the script actually sets ATOM_DP_LB_REQ_EQUIV=512 and STATE_CHECKPOINT_INTERVAL_TOKENS=8192 unconditionally — readers/auditors relying on the changelog to explain observed throughput will be misled about the real DEP config. Fix: correct the bullet to state ATOM_DP_LB_REQ_EQUIV=512 and an 8192-token checkpoint interval, matching benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh lines 65 and 57.
Extended reasoning...
perf-changelog.yaml lines 6958-6959 (the new dsv4-fp4-mi355x-atom-agentic-mtp entry) state 'zero request equivalence' and 'set the state checkpoint interval to 32768 tokens'. The script sets export ATOM_DP_LB_REQ_EQUIV=512 (line 65) and STATE_CHECKPOINT_INTERVAL_TOKENS=8192 (line 57) with no code path that changes either value to 0 or 32768. Since perf-changelog.yaml is append-only and treated as the authoritative record of what changed and why (per AGENTS.md invariant #1), this factual error will mislead future engineers debugging DEP throughput/latency differences from the documented values.
Verification: nit. The changelog claim and the script genuinely disagree. perf-changelog.yaml line 6958 (new dsv4-fp4-mi355x-atom-agentic-mtp entry) states: "...use least-tokens routing with session affinity and zero request equivalence, set the state checkpoint interval to 32768 tokens...". The script benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.sh sets STATE_CHECKPOINT_INTERVAL_TOKENS=8192…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
4 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34311990815 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34388489723 |
为 DSV4-Pro 的 DPA/DEP 路径补齐小 batch CUDA graph，避免非二次幂 decode batch 向上填充造成的 attention、MoE 和 RCCL 无效计算。保留 TP 路径默认行为，并为高并发配置保留大 batch graph。
Dense DEP decode graphs / DEP decode 稠密图优化Added commit On the current PR configuration (8K state checkpoint, default scheduler controls), a controlled C64 900-second A/B changed only the graph ladder and kept synthetic acceptance at 2.49:
The official C64 server log shows that 2,140 of 3,270 scheduler samples (65.44%) have non-power-of-two running-request counts, most commonly 5/6/7, so the gain matches the observed padding mechanism. Cost: about +13 seconds startup and +0.12 GB graph memory per GPU. Correctness was also checked with real MTP on all 1,319 GSM8K samples: flexible/strict exact match = 95.15% / 95.22%, with 65.95% MTP acceptance; no accuracy regression versus 94.69% / 94.84% before the graph change. 提交 在当前 PR 配置（8K state checkpoint、默认调度参数）上做了 900 秒 C64 隔离 A/B，唯一变量是 graph ladder，synthetic acceptance 同为 2.49：
官方 C64 日志的 3,270 个调度采样中，有 2,140 个（65.44%）是非二次幂 running-request 数，最常见为 5/6/7，和减少 padding 的机制一致。代价约为启动增加 13 秒、每张 GPU graph 显存增加 0.12 GB。 精度也使用真实 MTP 跑完 1,319 道 GSM8K：flexible/strict exact match 为 95.15% / 95.22%，MTP acceptance 65.95%；相对改动前 94.69% / 94.84% 没有精度回退。 |
3 participants
Summary / 概要
Switch the DeepSeek-V4-Pro MI355X ATOM AgentX concurrency 48/64/96/128/256 band introduced by perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP #2778 from DPA+TBO with the default MoE transport to native RCCL DEP: TP8, DPA8, EP8, RCCL all-to-all, TBO off, and EPLB off.
Keep the concurrency 1/2/4/8/16 TP band unchanged at TP8/EP1 and retain its golden MTP acceptance length.
Pin
rocm/atom-dev:nightly_202609071454(sha256:f252ec9a30e3d17b91f15887ea32baa1a0c5503b775f5adf42ff7cee8cfff769), built by the successful ROCm/ATOM release run at merge commit5a9c2068containing native RCCL DEP.Match the validated local c48 run: least-tokens scheduling, session affinity,
ATOM_DP_LB_REQ_EQUIV=0, 32K state checkpoints, terminal-MTP optimizations, real MTP acceptance, and the validated AIPerf timeout/failure settings.将 perf(amd): add DP-attention band to DSV4 ATOM MI355X AgentX MTP #2778 引入的 DeepSeek-V4-Pro MI355X ATOM AgentX c48/64/96/128/256 档位，从默认 MoE 通信的 DPA+TBO 切换为原生 RCCL DEP：TP8、DPA8、EP8、RCCL all-to-all，并关闭 TBO 和 EPLB。
保持 c1/2/4/8/16 档位为原来的 TP8/EP1，同时保留该档位的 MTP golden acceptance length，不改变低并发测试语义。
固定使用
rocm/atom-dev:nightly_202609071454（sha256:f252ec9a30e3d17b91f15887ea32baa1a0c5503b775f5adf42ff7cee8cfff769）。该镜像由成功的 ROCm/ATOM 发布任务基于包含原生 RCCL DEP 的合并提交5a9c2068构建。对齐本地已验证的 c48 参数：least-tokens 调度、session affinity、
ATOM_DP_LB_REQ_EQUIV=0、32K state checkpoint、terminal-MTP 优化、真实 MTP 接受率，以及已验证的 AIPerf 超时和失败阈值。Local validation / 本地验证
Validated source run:
atom-native-rccl-dpa8-ep8-eplboff-agentx-c48-ci3600-realmtp-fp4idx-r4-20260902本地 c48 实跑完成 3,920 个请求且请求错误数为 0；输出吞吐为 1,066.71 token/s，prompt cache-read rate 为 96.55%，ITL 中位数为 19.12 ms。
Test plan / 测试计划
bash -n benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.shParse
configs/amd-master.yaml,configs/runners.yaml, andperf-changelog.yamlwith PyYAMLgit diff --checkGenerate the exact
dsv4-fp4-mi355x-atom-agentic-mtpmatrix withgenerate_sweep_configs.pyVerify TP points are c1/2/4/8/16 with EP1 and DEP points are c48/64/96/128/256 with EP8+DPA
Bash、YAML 和补丁格式检查通过
仓库矩阵生成器成功展开全部十个测试点，并确认低并发 TP 档和高并发 DEP 档的边界正确
References / 参考
Note
Low Risk
Changes are limited to benchmark recipes, sweep YAML, and changelog; they do not alter application runtime code paths outside the documented ATOM serve command for this config.
Overview
Moves the DeepSeek-V4-Pro FP4 MI355X ATOM AgentX high-concurrency sweep (c48–256) from DPA + TBO with default MoE transport to native RCCL DEP (TP8 / DPA8 / EP8), pins
rocm/atom-dev:nightly_202609071454, and keeps the low-concurrency TP-only band (c1–16, EP1) on golden synthetic MTP acceptance while DEP and eval use real acceptance.The benchmark script adds config-driven expert parallel, DEP server flags (
--all2all-backend rccl, least-tokens load balance, standard MoE), enforcesEP_SIZE=TPfor DEP, drops TBO/NUMA queue knobs, and for DPA runs supplies exact--cudagraph-capture-sizesfor small decode batches (plus 256/512 whenmax-num-seqs> 128).perf-changelog.yamlrecords the image/DEP migration (#2886) and the CUDA-graph tuning with reported C64 A/B gains while keeping an 8192-token state-checkpoint interval.Reviewed by Cursor Bugbot for commit 4532d8f. Bugbot is set up for automated code reviews on this repo. Configure here.
This PR supersedes #2886. The commit history is unchanged; the head branch has been moved from the personal fork to SemiAnalysisAI/InferenceX as requested by the maintainers.